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ABSTRACT: 

Reinforcement schemes are a class of non-Markovian stochastic processes. Their non-Markovian 
nature allows them to model some kind of memory of the past. One subclass of such models are 
those in which the past is exponentially discounted or forgotten. Often, models in this subclass have 
the property of becoming trapped with probability 1 in some degenerate state. While previous work 
has concentrated on such limit results, we concentrate here on a contrary effect, namely that the 
time to become trapped may increase exponentially in 1/x as the discount rate, 1 — x, approaches 1. 
As a result, the time to become trapped may easily exceed the lifetime of the simulation or of the 
physical data being modeled. In such a case, the quasi-stationary behavior is more germane. We 
apply our results to a model of social network formation based on ternary (three-person) interactions 
with uniform positive reinforcement. 
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1 Introduction 



Stochastic models incorporating mechanisms by which likelihoods of outcomes increase according to 
their accumulated frequencies have been around since the introduction of Polya's Urn EP23 . The 
mathematical framework for many of these models appears in the literature on stochastic approx- 
imation, beginning with |RM51| . in the urn model literature Fr65 , in the literature on reinforced 
random walks PemSflJ IDav90l ILim01| . and in the literature on the relation between stochastic 
systems and their deterministic mean-field dynamical system approximations BH95l lBen96| . 

These processes, known in the mathematical community as reinforcement processes, have long 
been used by psychologists as models for learning [BM55I I1T69I INor72l [Lak81 . Increasingly, re- 
inforcement models have been adopted by other social scientists as interactive models in which 
collective learning takes place in the form of network formation or adaptation of strategies: so- 
ciologists studying the "small world" network phenomenon WS98 , B WOO , formation of dyads of 
reciprocal approval FM96 ; economists studying evolutionary game theory |MS82| , strategic learn- 
ing |RE95| |FK93| or its interaction with network structure E1193, AI97 . These models are designed 
to explore mechanisms by which agents with limited information, rationality or sophistication may 
nevertheless achieve advantageous social structures via the application of simple rules for behavior 
change. 

Due, perhaps, to a dearth of types of simple local rules (or perhaps to a lack of imagination 
on the part of modelers), most reinforcement models fall into one of two classes. The first class 
contains models for which the past is weighted uniformly. This class includes the urn models, 
stochastic approximations and reinforced random walks mentioned above, as well as the economic 
game theory models of Erev and Roth jRE95| . Uniform weighting means that the step from time 
n — 1 to n represents a fraction of only l/n of the total learning up to time n, so one obtains a 
time-inhomogenous process in which the hidden variables change by amounts on the order of l/n 
at time n. The second class consists of models in which the past is exponentially discounted or 
forgotten. This class includes the learning models of the 1960's and 70's [1X69] . as well as many 
contemporary models of repeated economic games, e.g. [BL02 . In these models, the weight of the 
present is asymptotically equal to the discount parameter, x, defined as the x for which an action 
t units of time will be weighted by (1 — x) . More precisely, the fraction of total learning between 
time n — 1 and n of the total learning to time n will be roughly the maximum of l/n and x. 

Our chief concern in this paper is to study how the discounted process approaches the non- 
discounted process as x — ► 0. The long-run behavior in these two cases is qualitatively different. 
Limit theorems for non-discounted processes have been obtained chiefly in the framework of stochas- 
tic perturbations to dynamical systems. Typically, the stochastic system converges to limit points 
or limit cycles of the dynamical system that corresponds to the mean motion of the stochastic sys- 
tem |BH95| . The random limit is supported on weakly stable equilibria jPem90] . though the system 
may remain near unstable equilibria for long periods of time (see |BP03| for a discussion of this 
phenomenon in continuous time, and |PS03b) for a case study in discrete time). 

In the discounted processes we study here, there are trapping states, into which the chain must 
eventually fall (there is another kind of discounted process we are not concerned with here, which 
converges to an ergodic Markov chain, see, e.g., |IT69 ). The reciprocity model of |BL02j . for 
example, is of this type. From the point of view of studying the transition as discounting goes to 
zero, the most interesting case is when the trapping states are disjoint from the stable equilibria in 
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the non-discounted process. Trapping states for the discounted process must always be equilibria for 
the non-discounted process, but when all the trapping states are unstable equilibria (equivalently, 
all the stable equilibria of the non-discounted process are non-trapping in the discounted process), 
then the conflict between the discounted and non-discounted behavior is maximized. 

The transition is easy to describe informally. As the discount rate approaches zero, the discounted 
process behaves for a longer and longer time like the non-discounted process, and then abruptly falls 
into a trap. Of course when the discounting parameter is x it takes time at least on the order of 
1/x for the system to notice there is discounting going on. But in fact, due to the learning that has 
gone on during this phase, it will take time of order exp(ca; _1 ) before the system discovers a trap 
and falls in. It is in fact not hard to guess this via back-of-the-napkin computations. One of our 
main motivations for pursuing this rigorously was to explain why simulation data contradicted the 
easily proved limit theorem: it was because the time scale of the simulation (let alone of any real 
phenomenon modeled by the simulation) was never anywhere near the time needed to find a trap. 

Our purpose in the present paper is to prove various versions of this. In the next section, we 
present the ternary interaction model which was our original motivation for this study. Section 
then introduces a simple process that is a building block for the ternary interaction model. For that 
process, results about trapping times can be proved with the correct constant. The last section then 
proves a exp(ca; _1 ) waiting time result for a general class of models, but without the correct value of 
c. With a little linear algebra, this is shown to apply to the ternary interaction model of Section [3 



2 Three's Company: a ternary interaction model 

The following process is described in |PS03a| . where it is called Three's Company, and is put forth 
as a model for formation of ternary collaborations in a three-player version of Rousseau's stag 
hunting game. Fix a positive integer N > 4, representing the size of the population. For t > and 
1 5; i,j < define random variables W(i,j, t) and U(i,t) inductively on a common probability 
space (f2, P) as follows. The W variables are positive numbers, and the U variables are subsets 
of the population of cardinality 3. One may think of the U variables as random triangles in the 
complete graph with a vertex representing each agent. The initialization is W(i,j, 0) = 1 for all 
i ^ j, while W(i,i,0) — 0). The inductive step, for t > 0, defines probabilities (formally, conditional 
probabilities given the past) for the variables U(i,t) in terms of the variables W(r,s,t), r,s < N, 
and then defines W(i, j, t + 1) in terms of W(i,j, t) and the variables U(r, t), r < N. The equations 
are: 

N 

W(i,j,t+1) = (l-x)W(i,j,t)+J2 1 i,jeu(r,t)- (2-2) 

r=l 

Here (1 — x) is the factor per unit time by which the past is discounted, and the tr-field conditioned 
on is the process up to time i, 

T t :=a{W(i,j,u) :u<t}. 
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We may think of the normalized matrix 



W t : 
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as the state vector, which is then an asymptotically time-homogeneous Markov chain, with an 
evolution rule of the well known form 



where in this case, g(t) = 1/x + 0(l/t), the drift vector field fi maps the simplex of normalized 
matrices into its tangent space and may be explicitly computed, and £t are martingale increments 
of order 1. 

These equations model a social interaction in which each agent i at each time t invites two 
others to frolic 4 . For each agent i, the trio chosen by i is chosen from all possible trios containing 
i, according to the products of the weights W(i,-,t), Thus the probability of agent i forming the 
trio {i,j,k} is proportional to W(i, j, t)W(i, k, t). After the frolicking, fond memories ensue: each 
of the three pair weights W(i, j, t), W(i, k, t) and W(j, k, t) is increased by 1, and a portion x of the 
past weights is forgotten. For ease of bookkeeping, the weights of unordered pairs are defined as 
symmetric weights of ordered pairs, so the weights W(j,i,t),W(k,i,t) and W(k,j,t) are increased 
as well. We write W(e,t) for W(i,j,t) when e is the edge (unordered set) {i, j}. 

It is shown in PS03a, that the network always breaks into small cliques, with interactions 
occurring only among the cliques: 

Theorem 2.1 In Three's Company, with population size n > 4 and any discount rate x £ (0,1), 
with probability 1 the population may be partitioned into subsets of sizes 3, 4 & n d 5, such that each 
member of each subset chooses each other with positive limiting frequency, and chooses members 
outside the subset only finitely often. Every partition into sets of sizes 3, 4 an d 5 has positive 
probability of occurring. □ 

Simulation data is also given there. For N — 6, if x = .4 (a rather steep discount rate), the 
network always breaks into two cliques of size 3, as predicted by the theorem. When N = 6 and 
x = .2, which is still a greater discount rate than one finds in most economic models, one finds, with 
runs of several thousand, that no such structure emerges. Instead, all six members of the population 
remain well connected. This is because of the exponential time scale of the transition from stable 
equilibria (well-connectedness is a stable equilibrium of the non-discounted model) to trapping states 
(two cliques of size three is the unique trapping state of the N = 6 discounted model) . Specifically, 
in the last section of this paper we will prove: 

Theorem 2.2 In the game Three's Company, for each N > 6 there is a 5 > and numbers cjv > 
such that in Three's Company with N players and discount rate 1 — x, the probability is at least S 
that each player will play with each other player beyond time exp(cAra; _1 ). 

4 Engage in some rewarding interaction such as anti-competitive price fixing 



E (W 1+ i - W t | T t ) = g(t) [ M (W t ) + £t] 



(2.3) 
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3 Trapping in one-dimensional discounted reinforcement 



In this section we analyze a one-dimensional process in which sharp quantitative results may be 
obtained on the exponential rate at which the time until trapping increases with 1/x, where 1 — x is 
the discount factor. This is in keeping with our philosophy of providing sharp results on a collection 
of simplified models that constitute building blocks for more complicated models. In the last section 
we will apply the principles gleaned from this to get bounds on the exponential rate of increase of 
trapping time in the Three's Company model. 

Let us consider a system whose state vector varies in the interval [0, 1] with evolution dynamics 
that are symmetric around an attractor at 1/2, and whose transitions from state w have a profile that 
depends on w and is scaled by x. In analogy with models such as Three's Company, we assume that 
the unsealed transitions have variance bounded from below as w varies over compact sub-intervals 
of (0, 1). Thus the rules of evolution of the state vector W may be given in terms of probability 
distributions Q w , parametrized by w € [0, 1], with bounded support, satisfying Q w (s) = Qi- w (—s), 
and obeying: 

P(W(n + 1) - W(n) ex-S \ T n ) = Q w{n) (S) (3.1) 

on the event that W(n) is in a compact subinterval I x , with I x f (0, 1) as x — » 0. We assume that 
the mean of Q w is positive on (0, 1/2) and negative on (1/2, 1), but that Q w has both positive and 
negative elements in its support and varies smoothly with w. 

As an example, one may consider a class of two-color urn models generalizing Friedman's 
Urn |Fr49l IFr65| in the discounted setting. An urn begins with R(0) red balls and B(0) black 
balls. At the n th time step, a random number U (n) of red balls and V(n) black balls are added to 
the urn. Conditional on the past, F(U(n) = k) = u w ^ n _ 1 - ) (k) and P(V(n) = k) = wi-w/(„_i)(fc), 
where W(n) := R(n)/(R(n) + B(n)) is the state parameter, in this case the proportion of red 
balls, and u w are probability distributions on the nonnegative integers, continuously varying in the 
parameter w € [0, 1], satisfying 

^2 k ku w (k) 1 
> w for < w < 



^2 k k{u w (k) +ui- w (k)) 2' 

At the end of each step, all balls are reduced in weight by a factor of 1 — x. For greater specificity, 
one may keep in mind an example where two balls are sampled: if they are of the same color then 
one ball of that color is added; if they are of different colors then one ball of each color is added. 

In the non-discounted system, where the step size scales as 1/n at time n instead of holding 
constant at x, the system is well approximated by a diffusion with incremental variance of order 
n~ 2 and drift n -1 /!^, with fi w being the mean, Q w , of Q w . Thus (|2.3|) holds with g(t) = t~ x and 
ju(if) = Qw The system must converge to the unique attracting equilibrium at 1/2 Pe m90| . In the 
discounted case, although the state must converge to or 1, the logarithm of the expected time to 
come near or 1 may be computed in terms of the following data. 

Pick any w G (0, 1/2). The quantity 

Z w {\) := / exp(-Xy)dQ w (y) 



is equal to 1 at A = 0. The derivative (d/dX)Z w (X)\\ = o is given by J (—y) dQ w (y), which is negative 
by the assumption that Q w has positive mean. On the other hand, since Q w gives positive probability 
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to negative values, we see that as A — > oo, Z w — > oo, and by convexity of Z w {-) it follows that there 
is a unique A„j > for which Z W (X W ) = 1. Define 

,1/2 

A(u>) := / A u dit 

and let C := A(0). 

Theorem 3.1 Let I x f (0, 1) as 1 — x f 1, slowly enough so that transitions outside of [0, 1] are 
never possible. Let T x be the expectation of the first time n that W(n) I x . Then as 1 — x | 1, 

xlogET^ -> C. 

Remark: This is essentially a large deviation problem, so the rate C is not determined by the mean 
and variance of Q w but rather by the exponential moments of Q w . In particular, there are many 
processes which satisfy (I2.3|) with the same g and fi, but their large deviation rates depend on the 
fine structure of the increment distribution through the exponential moments, as captured by Z w 
and X w . The solution of this rate problem is standard; a similar analysis may be found, for example 
in JB7M Section 5.8.2]. 

Proof: For one inequality, we fix any 5 > 0. Define the quantity 

M (s) (t) := exp((l - S)x- 1 A(W{t)) . 

Since Q w varies smoothly with w and has bounded support, we see that as x — > 0, 

A(W(t + 1)) - A(W(t)) = (W(t + 1) - W(t))(-X w(t) + 0(x)) . 

Therefore, since the conditional distribution of x~ 1 (W(t + 1) — W(t)) given Tt is given by Qw(t): 
we see that 

E(M (a) (t + 1) | T t ) = M (s) {t) E [exp((l - 5) X - 1 (W(t + 1) - W(t)){-X w(t) + O(x)))] 

- M (s) (t)Z w(t) ((l-S)X w{t) ) (3.2) 

uniformly in W(t) as x — > 0. We know that Z w {-) < 1 on (0, A^,), hence we may pick x = x(5) small 
enough so that 

M^^-^M^^ + l)!^) < 1 
or in other words, so that is a supermartingale. 

Let 7a; = [a Xl 1 — a x ]. Starting with W(to) E (1/2 — <5, 1/2), and stopping at the time r when 
W^(-) exits [as, 1/2], we have for some constant c(8) going to zero with 5, 

exp(s- 1 c(5)) > M (a) (* ) 

> E(M w (r)|^ t0 ) 

> P(M W (r) < a,) exp^ 1 A(a x )) 

which implies that 

logP(M (4) (T) < o a ) < -(1 -<5)x- 1 (A(0) + o (l)) 
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as x — * 0. A completely analogous argument shows that the process started in (1/2, 1/2 + 5) exits 
[1/2, 1 — a x ] with at most this probability as well. The trajectory of W(-) may be decomposed into 
segments that begin in [1/2 — (5,1/2 + S] and end when W(t) — 1/2 changes sign or W(t) £ I x . 
We have shown that the expected number of trajectories is at least exp(a; _1 (l — <5)(A(0) + o(l))) 
as x — > 0, which implies that the number of time steps until exiting I x is at least this great, once 
W(t) G [1/2 — 6,1/2 + 5]. Letting c'{8) denote the probability of entering this interval, we see that 

ET X > c'(S) exp(x _1 (l - <5)(A(0) + o(l)) 

as x — > 0, and finally, sending S to zero proves that 

lim info; log ET X > C. 



For the other direction, define a tilted measure on the space of trajectories {W(t) : t = 0, 1, 2, . . .} 
as follows. The equation (|3.1|l is replaced by 

P(W(n + 1) - W(n) ex-S \ T n ) = Q w[n) (S) 

where S > is fixed and the Radon-Nikodym derivative is given by 

dQ W ( , = exp[(l + S)(A(w + y)-A(w))] 

dQ w [V) J exp [(1 + S)(A(w + y)- A(w))] dQ w {y) " 



The measure P is designed to have two properties. First, the process {W(£)} is a supermartingale 
on [a x , 1/2] with respect to P for sufficiently small x. To see this, note that this is equivalent to Q w 
having negative mean, which is equivalent to 



ye 



l+«)(A(»+»)-A(«)) dQ^y) < 



for all w G [a x ,l/2\. The quantity A(w + y) — A(w) is equal to y(— X w + o(l)) as x — > 0, so it suffices 
to show that 

" ye -(l+S)y\ m d Q w (y)<0. 

But this follows from the fact that Z w is convex and increases through 1 at X w : the derivative at 
(1 + 6)\ w must therefore be positive, and the derivative may be identified as 



proving the supermartingale property. 



The second property is that if r is the exit time of [a x , 1/2], then on the cr-field J- T , dP/dP is at 
most exp((l + 5)x~ 1 A(0)). Indeed, by its definition, 

— (W(t ) 1 ...,W{T))=[[ 



dP y vuy '"- ! K " lJJexp{(l + S)(A(W(t)+y)-A(W(t)))}dQw(t)(y)' 
The denominator of each factor is at least 1 by the fact that 

M ( _ 4) (t) := exp((l + 8)x- 1 A(W(t))) 
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is a submartingale when x(5) is small enough, which is proved by a computation exactly analogous 
to (|3.2() . The product of the numerators is simply exp((l + 5)x~ 1 A(W(t)) — A(W(to))), which is at 
most exp((l + <5)x _1 A(0)), proving the second property. 

Running P on [a x , 1/2] and its reflection on [1/2, 1 — a x ], the process 1/2 — |l/2 — W(f)| is a 
supermartingale with incremental variance of order x~ 2 . The median time for it to reach a value 
less than a x is therefore at most 0(x~ 2 ). Comparing P and P, we find that there is a c such 
that from any starting data, the probability of exiting I x by time cx~ 2 is at least (l/2)(dF/dF) > 
(1/2) exp(— (1 + S)x~ 1 A(0)). Breaking into time intervals of size cx~ 2 , it then follows that the mean 
time to exit I x is at most 2cx~ 2 exp(C(l + S)x~ 1 ). As this holds for any 5 > (and constants 
depending on <5), this proves that 

lim sup x log ET X < C 

and finishes the proof of the theorem. □ 



4 Proof of Theorem EH 

In analogy with the one-dimensional toy model, we expect to find an exponential wait to trapping in 
Three's Company if the non-discounted system has an attractor outside of the limit set of absorbing 
states of the discounted system. Unfortunately, at this point we cannot see any way to compute 
the large deviation rate in multi-dimensional problems. The standard multi-dimensional analogue 
to Theorem 13. II is expressed as a variational result involving minimizing a functional over all paths. 
We settle for proving the existence of a nonzero exponential rate in x . The following result will 
imply Theorem 12. 21 

Proposition 4.1 Let the vector-valued Markov chain W(t) satisfy 

P(W(t + 1) - W(t) e xS | W{t)) = Q m (S) (4.1) 

with Q w having bounded support and varying smoothly as w varies over some closed neighborhood J\f 
of a point c. Suppose there is a strong Lypunov function V , meaning that V is smooth and bounded 
with V < on J\f c , V(c) > and 

J V(w + y) dQ w (y) > V(w) 

for all w G AT. Then there is a constant 7 such that for all h in some smaller neighborhood of c, 

E n T x > 7 exp( 7a ;- 1 ) 
for sufficiently small x, with T x being the time to exit N ' . 

Proof: Given a non-negative parameter, A, define M(t) = M\(t) — exp(— \V(W(t))). Arguing as 
in the first half of the proof of Theorem 13. II we see from the bounded support hypothesis that for 
fixed uieJV, 

Af A (t)- 1 E(Af A (i + 1) - M\(t) I T t ) (4.2) 
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vanishes at A = and has negative derivative. By compactness of TV and smoothness of Q w , we may 
choose a A > so that l|4.2() is negative for all w £ TV, implying that M(t) is a supermartingale up 
to the exit time of TV. Let TV' be the neighborhood {h e TV : V(h) > V(c)/2}. Using T G to denote 
the first time r > that W{t) E G, we then have, for V(h) > V(5)/4, 

e -AV(c)/4 > E K (M>(0)) 

> P K (7Vc<7V). 

Breaking the time Ta/<= into sojourns away from TV' then proves the theorem with 7 = AV(c)/4. □ 

In Three's Company, if we start with the sum of the weights equal to 3TVx _1 (that is, in sta- 
tionarity), then the dynamics are described exactly by 14.111 . We need only check the existence of a 
strong Lyapunov function. This will follow if we can identify a hyperbolic attractor for the vector 
field F(-), where F(w) is the mean of Q w . Indeed, if F vanishes at a point c and dF(c) has eigenval- 
ues with negative real parts, then there is a quadratic function V near c satisfying \/V ■ F > which 
we may take as the Lyapunov function. All that remains is to identify the hyperbolic attractor for 
the mean motion field. 

The mean motion is given by a vector field F on the state space. The state space is the set of 
non-negative real functions X on the edges summing to 3iVx _1 , which we think of as embedded in 
the cone of non-negative functions, since F extends naturally via F(XX) = F(X). The computations 
are a little more convenient when we normalize the sum of weights to be (^). It is also convenient 
to let n = N — 1 be one less than the number of agents. The attractor on which we focus is the 
symmetric point c defined by c(e) = 1 for all e. It is immediate to verify that F(c) = 0. In order 
to verify that c is an attractor for F, we need to compute the differential of F at c. Accordingly, 
let l e denote the function that is 1 on e and elsewhere. The derivative of F in the l e direction is 
computed as follows. 

Let the edge weights at time t be given by l+el e - The expected number of i for which / e U(i, t) 
is — + 0(e) for all /. By symmetry, the 0(e) term depends only on whether / shares two, one or 
zero endpoints with e. For example, in the case / = e, we compute the expected number of times e 
is reinforced as follows. Let e = {v,w}. Then 

P(e e U(v, f)) = (n - 1) ; L + £ • 
(2) + ( n - l ) e 

The probability of e G U(w,t) is the same. For z 7^ v,w, the probability of e 6 U(z,t) is exactly 
(2) • Summing yields an expected increment in W(e,t) of 

n/ 1 \ l + € n- 1 6 4(n-2) n . y2 

2 Tl - 1 7^ ? 7T + -T^r = -+ 1 e + ° e ■ 

+"" l£ 2 n n 



We write this as 



6 / , 2(»-2) , 2 , 



nl 1 + V f + °( e » 



Computing, the other two expectations in this manner, we find that the expectation for / at distance 
j from e is ^(1 + Bje + 0(e 2 )), where 

3n 
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B 1 = 
B 2 = 



3n(n — 1) 



From this it follows that for any h, 



1 1 —>—>—> 
-E(W(t + 1) I W(t) = c + eh) = c + Mh + 0(\h\ 2 ) 
6 

where M is a generalized circulant matrix (symmetric under the action of edge permutation on pairs 
of edges) with entries Bq on the diagonal, B 2 for disjoint edges, and zero otherwise. Since F is the 
vector field pointing toward E(M / (t + 1) — W(t) \ W(t)), the differential of F is, up to the constant 
multiple ^, equal to M — I. 

The eigenvalues of a matrix such as M are particularly easy to evaluate, using the rubric of 
association schemes (see |Ter96l Section 2.2], which is taken from BI84 BCN89 ). All such matrices 
are elements of the Bose-Mesener algebra M. which, in the case of the incidence graph for edges of 
the complete graph, is commutative semi-simple of dimension 3. This implies that M has at most 
three distinct eigenvalues. These may be found by computing the action of M on the three shared 
eigenspaces common to all elements of M.. 

The null eigenspace has dimension 1 : M h — if and only if h = Ac. The other two eigenvalues 
may be gotten by choosing an edge e and setting the eigenvectors equal to a 2 H2 + a\H\ + agHo, 
where H 2 = l e , H± is the sum of 1/ over edges / sharing one vertex with e, and Hq is the sum of 
1 f over edges / disjoint from e. The action of M on such a sum produces another such sum, and is 
linear, having matrix 





3n(n — 1) 

with respect to a, b and c. The left eigenvectors of this are 

fn-l n-3 \ , { fn-l\ n-2 

1,1.1) . ^,-^,-1 and f , — ,1 



2 



The corresponding eigenvalues are 



2(n + l)(n-2) In 
, — ; : and 



n(n — 1) 3 n — 1 



The equations of mean motion are 

E(W(t + 1) - Wit) I Wit) - c + eh) = - X - l iM - I)h + 0{\h\ 2 ) , 

n 

whence the point c is attracting for sufficiently small x if and only if the real parts of all eigenvalues of 
M are less than 1. We have identified that this is so, and therefore c is attracting. The hypotheses of 
Proposition ^. II are therefore satisfied with a quadratic Lyapunov function, and Theorem 12 . 21 follows 
from Theorem l3.ll □ 
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